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A VIDEO BASED HANDWRITING RECOGNITION SYSTEM AND METHOD 

BACKGROUND OF THE INVENTION 
Field of the Invention 

[0001] The present invention relates generally to handwriting recognition systems and methods 
and, particularly to handwriting recognition systems and methods for pervasive devices. 

Description of the Prior Art 

[0002] One of the most promising methods of interacting with small portable computing devices 
such as Personal Digital Assistant (PDA), smart phones, and like wireless or wired pervasive 
devices, is by handwriting recognition. In contrast to the automatic processing of forms or postal 
addresses, where an image of the handwritten text is acquired and processed off-line, in 
handwriting recognition, the movement of pen is captured directly during the writing process 
(real-time). In order to obtain this "on-line" handwriting data, however, special input devices 
including touch sensitive pads together with special "smart" pens with built-in intelligence for 
obtaining and processing tilt information are necessary. 

[0003] The quality of the recognition process is very important for the acceptance of handwriting 
recognition as a means of input. In some cases, imperfect quality of recognition may be just an 
inconvenience, whereas in other cases, it may preclude usage of the technology at all. Therefore, 
any way to improve upon the quality of recognition is welcome. Furthermore, one that improves 
upon the quality of recognition at a low cost is advantageous as these devices become much 
more attractive to cost conscience consumers. 

[0004] One of the applications of handwriting recognition today is signature recognition, most 
typically for retail applications, and the like, etc. In these applications, the underlying method is 
based upon relatively smart and expensive pen device having intelligence for generating output 
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parameters such as tilt etc., which are additionally utilized in the recognition algorithms 
employed by the pervasive device. When a typical pervasive device such as an average PDA is 
used, typically the only pen used is a common plastic stylus. Given the price of a PDA at about 
300$, for example, it is not cost effective to add an expensive smart pen at about 80$, for 
instance. 

[0005] It would thus be desirable to provide a simple stylus device that is able to obtain the 
precision and quality of recognition of smart pen devices. 

[0006] In conventional handwriting recognition, e.g., a hybrid handwriting recognition system, 
there are implemented steps of: responding to a handwriting input from a user, and dynamically 
generating time ordered, stroke information which are used to determine a most probable 
character that the dynamic, time ordered stroke information is intended to represent. Such a 
system is described in a reference U.S. 6,01 1,865 entitled "Hybrid on-line handwriting 
recognition and optical character recognition system." 

[0007] An exemplary system 10 currently in use for handwriting recognition, e.g. for retail 
applications, is now described with respect to Figure 1 that depicts an external smart pen device 
1 5, a writing surface 1 8, and a handwriting recognition processing block 20 implementing 
recognition processes 22 running on a computer platform. Pen position information as well as 
other pen information such as tilt parameter information, etc., are extracted from the smart pen 
and writing surface 1 8 and transmitted through link 25 to an executing handwriting recognition 
process 22. The algorithm in use is on-line type of recognition using dynamic information from 
the pen. As mentioned above, price constraints simply will not permit usage of the relatively 
expensive smart pen device 15 and a handwriting recognition block 20 to run on pervasive 
platform such as a PDA. 

[0008] As mentioned, current PDA devices are equipped with a plastic stylus with no 
intelligence attached. The only function of the stylus is to press upon a touch sensitive PDA 
display surface to produce dynamic information. Figure 2 illustrates the conventional PDA 
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device 30 including a touch screen 32 for generating dynamic stylus handwriting information 
that is fed through an internal link 35 into handwriting recognition block 40 that runs on the 
same CPU as the rest of the PDA software. An obvious disadvantage of the approach of the 
system 30 depicted in Figure 2, is a lack of additional pen information such as tilt as compared 
with the system described with respect to Figure 1. 

[0009] In other types of recognition systems, a pure digital camera input may be used for the 
sake of recognition, however the processing and results may not be as precise. 

[0010] As many pervasive devices are increasingly entering the market equipped with an 
embedded digital camera of relatively high resolution (e.g., implementing VGA), it would be 
advantageous to utilize the digital camera for providing digital video data for use in the 
handwriting recognition process in such pervasive devices. 

[0011] It would thus be highly desirable to provide a handwriting recognition system and 
method for pervasive devices that includes a low cost digital video camera functioning to provide 
more handwriting information from a low cost stylus pen equipped with the device, and use this 
information together with pen position for the purpose of improving handwriting recognition. 

SUMMARY OF THE INVENTION 

[0012] It is an object of the present invention to provide a low-cost solution for improving 
handwriting recognition in pervasive devices. 

[0013] It is another object of the present invention to provide a low-cost solution for improving 
handwriting recognition in pervasive devices that includes utilizing a digital camera embedded in 
the device to obtain real-time information corresponding to movement of the simple stylus pen 
equipped with the pervasive device during use, and utilize this information together with stylus 
position information for the purpose of improving handwriting recognition. 
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[0014] It is a further object of the present invention to provide a pervasive device such as a PDA, 
smart phone, or other wireless or wired pervasive device that includes a low-cost solution for 
improving handwriting recognition, and particularly, a method and system that enables an 
embedded digital camera to obtain more information from the stylus pen used by the writer and 
process this information together with pen position for the purpose of improving handwriting 
recognition in applications for such pervasive devices. Such pervasive devices are equipped with 
processing capability including implementation of handwriting recognition algorithms that 
receive data input obtained from a digital camera embedded in the device in addition to 
commonly obtained pen stylus position information for the purpose of improving handwriting 
recognition. 

[0015] According to a first aspect of the invention, there is provided a system and method for 
use in a pervasive device that implements an embedded digital camera together with a simple 
stylus, and processing capability for generating additional information, such as tilt etc. This 
video information is processed by a particular algorithm that is able to extract tilt and/or other 
parameters needed for handwriting recognition. The camera is particularly installed in such a 
manner so as to capture a plane which is perpendicular to the PDA screen plane. The tilt 
extraction is possible by virtue of combining known pen geometry, digital camera output and 
touch screen output being coordinates of the pen. 

[0016] According to another aspect of the invention, there is provided a system and method for 
use in a pervasive device that implements an embedded digital camera together with a simple 
stylus, and processing capability implementing handwriting recognition algorithms, wherein 
handwriting recognition algorithms may be the same ones in use for retail applications. The only 
difference will be the source of tilt information - instead of getting this information from an 
external pen it will come from the camera. An image/pattern recognition algorithm is 
implemented to first process camera output data, e.g., low-frequency video information, extract 
pen tilt data, and feed it to a handwriting recognition algorithm that uses tilt to improve the 
quality of the handwriting recognition. 
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[0017] According to a further aspect of the invention, there is provided a pervasive device, such 
as a PDA, that implements an embedded digital (built-in) camera together with a simple stylus, 
and processing capability implementing handwriting recognition algorithms for providing 
precise handwriting recognition. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0018] Further features, aspects and advantages of the structures and methods of the present 
invention will become better understood with regard to the following description, appended 
claims, and accompanying drawings where: 

[0019] Figure 1 depicts a conventional external smart pen-based handwriting recognition system 
on a touch screen for use in retail applications; 

[0020] Figure 2 depicts a conventional PDA equipped with a stylus for use in handwriting 
recognition on a touch screen; 

[0021] Figure 3 depicts a PDA device equipped with an embedded digital camera utilized for 
generating information used in the handwriting recognition system and methodology of the 
invention; 

[0022] Figure 4 depicts a flow chart implementing the handwriting recognition system with 
additional information obtained from the embedded camera; 

[0023] Figure 5 depicts a PDA device implementing a touch screen, a simple stylus, and an 
embedded digital camera utilized for generating information used in the handwriting recognition 
system and methodology of the invention; 

[0024] Figure 6 depicts a geometry for generating stylus coordinates used in calculating stylus 
tilt information; and, 
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[0025] Figure 7 depicts a process sequence for calculating the tilt according to the methodology 
of the invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

[0026] The present invention is directed to a system and method for improving handwriting 
recognition for PDA device applications equipped with a plastic stylus. 

[0027] Figures 3 and 5 illustrate a PDA device 50 having a digital touch screen device 52 
including a writing surface 53 for a simple stylus pen 59, and a built-in digital camera device 55 
having functionality for generating dynamic video data output corresponding to stylus movement 
when manipulated by a user writing on the touch screen surface. A processing unit (CPU) of a 
PDA device or like pervasive device receives data obtained from the digital touch screen 52 and 
output from the digital camera device 55 and implements algorithms for performing handwriting 
recognition based on both types of information. The embedded digital camera 55 is preferably 
mounted in such a position and angle so as to capture video images along a plane that is 
perpendicular to the plane of touch screen surface 53 and of a range advantageous to capture 
handwriting anywhere on the touch screen surface. The conventional stylus pen 59 when used in 
a handwriting application will be located in such a plane. 

[0028] As shown in Figure 5, the PDA or like pervasive device 50 includes a digital camera 55 
capable of capturing images in a plane 60 perpendicular to screen 52 and by virtue of this, 
processes are implemented for dynamically calculating tilt information associated with the stylus 
movement during a user's handwriting and outputting the same for use in the handwriting 
recognition algorithm. In the following description, stylus tilt is shown to be integral in 
handwriting recognition. The same scheme can be applicable for other kinds of information to 
be extracted by means of digital camera and used in handwriting recognition algorithm. 
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[0029] As shown in Figure 6, the tilt angles, e.g. an angle theta "0", associated with the tilt of the 
stylus are calculated from the output video images of the stylus 59 along the planes as depicted 
in the figure. Particularly, Figure 6 depicts a defined coordinate system, e.g., an OXYZ system 
75, including an OYX plane representing the PDA touch screen surface 53 and, a ZOX plane 
"equal" to the camera image. Given that the lower end 58 of stylus (L) touches the PDA screen 
touch screen surface 53 and has coordinates Lx, Ly, (Lz = 0) and whereas an upper end 57 of 
stylus (U) includes coordinates Ux, Uy, and Uz, from the geometry of the stylus 59, the length of 
the stylus may be calculated according to equation (1) as follows: 

Length _Of_Stylus = sqrt( (Lx-Ux) A 2 + (Ly-Uy) A 2 + (Lz-Uz) A 2) ( 1 ) 

It is given that Lz=0 because lower end of the stylus L touches the PDA screen and, coordinates 
Ly and Lx are known from the screen controller device that controls touch screen data 
acquisition. Further, as a result of processing the digital video camera images, the lengths Ux 
and Uz may be extracted. Thus, simple geometry dictates that if Uy is known, then stylus tilt 
may be calculated. From equation (1), Uy may easily be determined given all this information. 
Thus, knowing Uy, the tilt angle of the stylus is easily calculated. 

[0030] Figure 7 depicts a process sequence 80 for calculating the tilt according to the 
methodology of the invention. In order to calculate the tilt, there are three stages as shown in 
Figure 7. At a first step 82, information from digital camera device is fed into a pattern 
recognition block 84 where a pattern recognition task is performed. The function of pattern 
recognition block 84 is to perform processing to extract stylus image from the video images 
captured by the embedded camera. To facilitate in this task, the simple plastic stylus is provided 
with recognition enhancements so that portions of the stylus may be easily recognized and data 
easily extracted for processing. For instance, as shown in Figure 5, the stylus may be designed to 
include various colored segments 42a, 42b to form a structure that is known to the algorithm of 
pattern recognition processing block 84. Once the stylus pattern is extracted, a third step 86 
calculates coordinates of the stylus, Ux and Uz as described herein with respect to Figure 6. 
Further, these coordinates together with the coordinates of the lower end 58 of the stylus 
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generated by the touch screen controller device 54 are processed in processing step 88 where 
actual stylus tilt information is computed. 

{0031] Figure 4 illustrates the whole data flow of the handwriting recognition process for a 
pervasive device: an embedded and specifically mounted digital camera 55 captures the stylus 
image and outputs it into a processing block 80 which is responsible for tilt computation as 
described with respect to Figure 7. As described, the tilt computation block 80 receives input 
from the touch screen controller 54 to obtain real-time coordinates of the stylus low end. The 
output of the tilt computation block/method 80 is fed into a further processing block 90 together 
with the stylus coordinate data output from the touch screen controller 54. The processing block 
90 performs the actual handwriting recognition task having inputs from both camera and touch 
screen controller. 

[0032] While the invention has been particularly shown and described with respect to illustrative 
and preferred embodiments thereof, it will be understood by those skilled in the art that the 
foregoing and other changes in form and details may be made therein without departing from the 
spirit and scope of the invention that should be limited only by the scope of the appended claims. 
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